Fractal-based decision engine for intervention

ABSTRACT

A method/apparatus/system for educational intervention based on a response metric is disclosed. The notice is generated in response to the collection of user and question data, the sending of questions, the receipt of answers, the evaluation of the correctness of the answers, the generation of a response metric, the comparison of the response metric to a threshold, and the generation of the report or notice. The response metric can be reflect the scatter, randomness, and/or slope of student provided answer data, and can be a fractal dimension.

BACKGROUND OF THE INVENTION

This disclosure relates in general to Learning Management Systems (LMS)and/or Online Homework Systems (OHS) and, but not by way of limitation,to assisting students using the LMS and/or OHS.

Student comprehension evaluation can facilitate providing students withrequired learning resources. In such evaluation, a student receives atask including one or several questions/prompts. In response to thesequestions/prompts, a student provides answers. These answers areevaluated to determine the number of questions/prompts that the studentcorrectly answered, which then, results in a grade or a score for thetask.

In more advanced learning environments, a student can interact with anLMS to receive educational training. The LMS can provide the studentwith tasks and can determine a score indicating the number ofquestions/prompts that the student correctly answered. These evaluationprocedures do not provide reliable detection and reliable earlydetection of student comprehension and thus do not meet the needs ofeducators or students. Currently, these lacking evaluation proceduresused in the LMS results in lost opportunity.

BRIEF SUMMARY OF THE INVENTION

In one embodiment, the present disclosure provides a learning systemincluding user devices, databases, and a managed learning environment,each of which are connected via a network. The learning system collectsquestion and user data and generates a questionnaire based on thecollected data. This questionnaire is provided to the student, and inresponse, answers to the questions are submitted by the student. Thelearning system determines which of the student provided answers arecorrect and applies a boolean-value to an answer based on whether thestudent correctly answered the question. This process is repeated for aplurality of the answered questions. The boolean-values, and/or a sum ofthe boolean-values are used to generate a response metric that canreflect the scatter, randomness, and/or slope of the boolean-valuesand/or the sum of the boolean-values associated with the studentprovided answers. The response metric is compared to a threshold, whichcan trigger the generation of a notice.

In another embodiment, the present disclosure provides a method fordetecting a threshold of scatter or randomness in questionnaire answerdata. The method can include the generation of a questionnaire, storinganswers to the questions in the questionnaire, providing thequestionnaire to a student, receiving answers from the student,determining whether the questionnaire answers are correct or incorrectaccording to a boolean-valued function, applying a boolean-value to thequestions according to whether they are correctly or incorrectlyanswered, generating a response metric based on the applied booleanvalues, comparing a response metric to a threshold, and providing anotice reporting whether the student surpassed the threshold.

In another embodiment, the present disclosure provides a method fordetermining and reacting to questionnaire response patterns. Electronicdata including questions and answers to the questions is stored. Thequestions are associated with a common topic. A user profile includingdata identifying a user and data relating to the user's past performancein answering questions is stored. Data defining a threshold value isstored. The threshold value is at least one of a value associated withthe user, a value associated with the topic, and a generic value.Questionnaire answer data is received. The questionnaire answer dataincludes user provided answers to the questions. A user associated withthe questionnaire answer data is determined. Questionnaire answer datais stored in an electronic store and it is determined whether thequestionnaire answers are correct or incorrect according to aboolean-valued function. A boolean value indicating a correct answer forcorrect user provided answers is stored. A boolean value indicating anincorrect answer for incorrect user provided answers is stored. Afunction on the boolean-value outcome to generate a response metricindicative of the scatter or randomness of the questionnaire answer datais performed. An educator for the user and the common topic associatedwith the questions is determined, which educator supervises the user'swork. The response metric indicative of the scatter or randomness of thequestionnaire answer data with the data defining the threshold value iscompared. A message to an educator identifying the user, the topic, andthat the threshold value has been reached is sent. Data is stored in theuser's user profile identifying the topic and indicating that thethreshold value has been reached.

In another embodiment, the present disclosure provides a learning systemfor determining and reacting to questionnaire response patterns. Thelearning system includes one or more hardware servers that areprogrammed to execute instructions. The one or more hardware servers areprogrammed to store electronic data comprising questions and answers tothe questions. The questions are associated with a common topic. The oneor more hardware servers are programmed to store a user profilecomprising data identifying a user and data relating to the user's pastperformance in answering questions, and are programmed to store datadefining a threshold value that can include at least one of a valueassociated with the user, a value associated with the topic; and ageneric value. The one or more hardware servers are programmed toreceive questionnaire answer data. The questionnaire answer dataincludes user provided answers to the questions. The one or morehardware servers are programmed to determine a user associated with thequestionnaire answer data, and are programmed to store the questionnaireanswer data in an electronic store. The one or more hardware servers areprogrammed to determine whether the questionnaire answers are correct orincorrect according to a boolean-valued function, to store a booleanvalue indicating a correct answer for correct user provided answers, andto store a boolean value indicating an incorrect answer for incorrectuser provided answers. The one or more hardware servers are programmedto perform a function on the boolean-value outcome to generate aresponse metric indicative of the scatter or randomness of thequestionnaire answer data. The one or more hardware servers areprogrammed to determine an educator for the user and the common topicassociated with the questions, which educator supervises the user'swork. The one or more hardware servers are programmed to compare theresponse metric indicative of the scatter or randomness of thequestionnaire answer data with the data defining the threshold value, tosend a message to an educator identifying the user, the topic, and thatthe threshold value has been reached, and to store data in the user'suser profile identifying the topic and indicating that the thresholdvalue has been reached.

In another embodiment, the present disclosure provides a method forgenerating a report in response to determining to questionnaire responsepatterns. Question data is received, which question data includesquestions, answers associated with the questions, and a topic associatedwith the questions. User data is received, which user data includes useridentification and user performance history. The user performancehistory indicates the number of questions that the user has received andthe number of questions that the user has correctly answered. Aquestionnaire is created based on the question data. The questionnaireis sent. Answer data is received, which answer data includes submittedresponses to the questions in the questionnaire. The correctness of theanswers is determined according to a boolean-valued function. Thecorrect answers are assigned a first boolean-value and incorrect answersare assigned a second boolean-value. A response metric based on theboolean-values assigned to the answers is generated. The response metricprovides an indicator of the degree of randomness in the answers. Thesubmitted responses are evaluated based on the response metric. Dataindicating degree of randomness in the answers is stored. A report ofthe results of the evaluation is generated. The generating of the reportincludes identifying the recipients of the report; and determining theuser associated with the report. The report is sent to the identifiedrecipients.

Further areas of applicability of the present disclosure will becomeapparent from the detailed description provided hereinafter. It shouldbe understood that the detailed description and specific examples, whileindicating various embodiments, are intended for purposes ofillustration only and are not intended to necessarily limit the scope ofthe disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is described in conjunction with the appendedfigures:

FIG. 1 depicts a block diagram of an embodiment of a learning system;and

FIG. 2 depicts a block diagram of an embodiment of a learning device;

FIG. 3 illustrates a block diagram of an embodiment of a LMS;

FIG. 4 illustrates a swimlane flowchart of an embodiment of a processfor detecting a threshold of scatter or randomness in questionnaireanswer data;

FIG. 5 illustrates a flowchart of an embodiment of a process fordetecting a threshold of scatter or randomness in questionnaire answerdata;

FIG. 6 illustrates a flowchart of an embodiment of a process for storingquestion and user data;

FIG. 7 illustrates a flowchart of an embodiment of a process fordetermining reporting information;

FIG. 8 illustrates a flowchart of an embodiment of a process fordetermining a threshold value;

FIG. 9 illustrates a flowchart of an embodiment of a process fordetermining a response metric;

FIGS. 10 and 11 illustrate flowcharts of embodiments of a process forgenerating a response metric;

FIG. 12 depicts a block diagram of an embodiment of a computer system;and

FIG. 13 depicts a block diagram of an embodiment of a special-purposecomputer system.

In the appended figures, similar components and/or features may have thesame reference label. Where the reference label is used in thespecification, the description is applicable to any one of the similarcomponents having the same reference label. Further, various componentsof the same type may be distinguished by following the reference labelby a dash and a second label that distinguishes among the similarcomponents. If only the first reference label is used in thespecification, the description is applicable to any one of the similarcomponents having the same first reference label irrespective of thesecond reference label.

DETAILED DESCRIPTION OF THE INVENTION

The ensuing description provides preferred exemplary embodiment(s) only,and is not intended to limit the scope, applicability or configurationof the disclosure. Rather, the ensuing description of the preferredexemplary embodiment(s) will provide those skilled in the art with anenabling description for implementing a preferred exemplary embodiment.It is understood that various changes may be made in the function andarrangement of elements without departing from the spirit and scope asset forth in the appended claims.

Referring first to FIG. 1, a block diagram of an embodiment of alearning system 100 is shown. The learning system facilitates 100learning by combining educational and student resources within aflexible system that can facilitate instruction and evaluation oflearning. The learning system 100 includes user devices 102, a managedlearning environment 104, and databases 106 that are connected, such asby a network 107, to send and receive information to provide educationalservices to a student.

The user devices 102 includes a student device 102-A, a teacher device102-B, an administrator device 102-C, and a parent device 102-D. Theuser devices 102 allow a user, including a student, a parent, and aneducator, including a teacher and an administrator, to access thelearning system 100. This access can be in the form of a studentreceiving educational material from the learning system 100, a studentreceiving a questionnaire having one or several questions from thelearning system 100, a student providing answers to a learning system100, a student, teacher, administrator, or parent receiving messagesfrom the learning system 100, a teacher selecting material for thestudent in the learning system 100, the creation of educationalmaterial, the receipt of information indicating a teacher's educationalperformance, the receipt of a class progress report, providinginformation to the learning system 100, receiving information from thelearning system 100, or any other similar activities. The details andfunction of the user devices 102 will be discussed at greater length inreference to FIG. 2 below.

The managed learning environment 104 is a collection of educationresources, tools, and software that can be accessed via the learningsystem 100. The managed learning environment 104 facilitates and managesthe education process by, for example, setting the parameters of alearning experience and the standardization of learning resources. Themanaged learning environment 104 includes a variety of components suchas a learning management system (LMS) 108, a content management system110, and an authoring tool 112. Although the managed learningenvironment 104 is depicted as including each of the above listedcomponents, the managed learning environment 104 may include more orfewer components than those depicted. Additionally, some or all of thesedepicted components can be located as capabilities and/or subcomponentswithin the other depicted components. In some embodiments, the learningsystem 100 can include an Online Homework System (OHS) instead of, inaddition to, or as a component of the managed learning environment 104,and can, like the managed learning environment 104, facilitate andmanage the education process by, for example, setting the parameters ofa learning experience and the standardization of learning resources.

The LMS 108, and similar educational components such as a virtuallearning environment (VLE) and a learning content management system(LCMS), provides for some or all of the administration, documentation,tracking, reporting, and delivery of education courses or trainingprograms. The LMS 108 facilitates education by allowing the use ofself-service and self-guided educational services so as to allow astudent to progress at their own selected pace and to access material asdesired. The LMS 108 also facilitates the personalization of content tothe student and enables knowledge reuse. The LMS is accessible by a userdevice 102 that is connected to the learning system 100 and having therequisite permission for such access. In some embodiments, the managedlearning environment 104 can include the OHS instead of, in addition to,or as a component of the LMS 108. In such an embodiment, the OHS canallow a user to access online material including education andassessment resources such as, for example, lessons, assignments,homework, tests, quizzes, or questions.

The content management system 110 allows the publishing, editing, andmodifying of content within the managed learning environment 104. Thecontent management system 110 also provides procedures to manage theworkflow within the managed learning environment 104. The contentmanagement system 110 provides the education content to the LMS 108,which content is delivered to the student or user by the LMS 108.

The authoring tool 112 allows the creation of content for use in themanaged learning environment 104 and the creation of submissions by thestudents. The content and/or submissions created by the authoring tool112 can be transmitted to the LMS 108 for use as part of an educationalcourse or training program, or for evaluation.

The databases 106 store information for use in the learning system 100.The databases 106 can be accessed by the user devices 102, by themanaged learning environment 104, by the components of the managedlearning environment 104, or by some or all of the other databases 106.The databases 106 can include, for example, a threshold database 106-A,a user database 106-B, a profile database 106-C, an educator database106-D, a topic database 106-E, and a question database 106-F. In someembodiments, the content of the databases 106 can be combined or dividedin the same way or in a different way than depicted.

The threshold database 106-A stores information relating to a threshold.The threshold is an indicator of student performance, and can, forexample, indicate the level of student performance such as excellentperformance, satisfactory performance, unsatisfactory performance, orany other desired performance level. The threshold can be a singlethreshold, or a plurality of thresholds. The threshold can be a generic,default threshold common to the learning system 100, the managedlearning environment 104, or the LMS 108, or the threshold can becustomized to a teacher, a program, a class, a grade level, an age, astudent, an aspect of one or several student profiles, a topic, or anyother desired metric.

The user database 106-B stores information relating to the users. Thisinformation can include the name of the user, a username, a password, orany other user identification information. This information can beprovided the users, including, for example, the student, the educator,or the parent.

The profile database 106-C stores user profiles. The user profiles canbe created by the users, or can be created based on the actions of theuser within the managed learning environment 104. The stored userprofiles can include information relating to education that the user hasreceived, past or current user performance, past, present, or futureuser courses, past, present, or future user teachers, past, present, orfuture user administrators, user parents, user preferences, any userdisability, or any other user characteristic.

The educator database 106-D stores educator information, such as aneducator profile. The educator profile can identify an educator, aneducator's past, current, or future course schedule, topics and/orcourses taught by an educator, educator evaluation information such as,for example, past or current student or class results, educatorpreferences, or other similar information.

The topic database 106-E can include topic information. This can includethe educational information relating to a subject matter, a course, atopic, or a division thereof. In some embodiments, the topic database106-E can include information dividing a subject matter into courses,dividing the courses into topics, and further dividing the topics intosubtopics. This division of the subject matter into smaller units can becontinued until the units have reached a desired size, such as, forexample, the size of a single lecture or portion thereof.

The question database 106-F can include question and associated answerinformation. The questions can correspond to and be associated withinformation stored in the topic database 106-E, and can thus relate to asubject matter, a course, a topic, or a division thereof. The questionscan likewise correspond to and be associated with information stored inthe educator database 106-D such as, for example, the teacher assigningthe questions to the student or the administrator responsible forsupervising the teacher. The questions can also be associated with ananswer so that responses received to the questions from students can beevaluated for accuracy.

An educator may access the learning system 100 via the teacher and/oradministrator device 102-B, and the educator may access informationstored in one or several of the databases 106. The educator may use thisinformation in connection with a component of the managed learningsystem 104 to view, create, modify, or evaluate educational material orstudent work product. Any changes to the educational material made bythe educator may be stored in one or several of the databases 106.

A student may access the learning system 100 via the student device102-A, and the student accesses educational information or student workproduct stored in one or several of the databases 106 via the managedlearning environment 104. Any changes and/or progress made the studentare tracked by the managed learning environment 104 and stored in one orseveral of the databases 106.

The student work product and/or any student progress is evaluated by acomponent of the managed learning environment 104 or by an educator. Thestudent's progress and/or evaluation is tracked by the managed learningenvironment 104 and is stored in one or several of the databases 106.The student's progress and/or evaluation is reported to the student, toan educator such as a teacher or an administrator, or to a parent.

With reference to FIG. 2, a block diagram of an embodiment of the userdevice 102 is shown. As discussed above, the user device 102 can be usedto access the learning system 100. Specifically, the user device 102 canbe used to access the LMS 108. In some embodiments, the user device 102accesses the LMS 108 via the network 107. In some embodiments, some orall of the components of the LMS 108 can be located on the user device102.

The user device 102 includes a user interface 202 that communicatesinformation to, and receives inputs from a user. The user interface 202can include a screen, a speaker, a monitor, a keyboard, a microphone, amouse, a touchpad, a keypad, or any other feature or features that canreceive inputs from a user and provide information to a user.

The user device 102 includes a network interface. The network interface204 communicates with other components of the learning system 100. Insome embodiments, the network interface 204 sends signals to and receivesignals from other components of the learning system 100 by, forexample, the network 107. The network interface 204 can communicate viatelephone, cable, fiber-optic, or any other wired communication network.In some embodiments, the network interface 204 can communicate viacellular networks, WLAN networks, or any other wireless network.

The user device 102 includes and education engine 206. The educationengine 206 allows the user to access the managed learning environment104 and can include components to allow a user to receive, create,and/or edit educational material. The education engine 206 can besoftware located on the user device 102, or a portal, such as a webportal, accessible via the user device 102.

The user device 102 includes a question engine 208. The question engine208 allows the user to receive and access questions associated witheducational material, and to provide answers to the received questions.The question engine 208 can include encryption and decryption componentsto allow the secure transmission of the questions. The question engine208 can further include features to track a student responding toquestions. This tracking can include, for example, gathering ofinformation relating to time spent on some or all of the questions orinformation relating to circumstances in which the questions werereceived, such as, for example, time of day, month, location, orconditions existing at the location. The question engine 208 can besoftware located on the user device 102, or a portal, such as a webportal, accessible via the user device 102.

The user device 102 includes a response metric generator 210. The metricgenerator 210 collects user provided answers and generates a responsemetric. This response metric is based on the number of correct answersprovided by the user and/or the number of incorrect answers provided bythe user. The details of the generation of the response metric will bediscussed at greater length below. The metric generator 210 can besoftware located on the user device 102, or a portal, such as a webportal, accessible via the user device 102. The metric generator 210 canalso be located on another component of the learning system 100 such as,for example, the managed learning environment, and accessible by theuser device 102.

The user device 102 includes an evaluation engine 212. The evaluationcan provide an evaluation of one or several user results, including, anevaluation of one or several student results, of the results of a class,or of an educator's effectiveness.

The evaluation engine 212 receives score information from the scoregenerator 210 and evaluates the user provided answers based on thescore. In some embodiments, this evaluation can include comparing thescore information with one or several threshold values to determine thelevel of performance. In some embodiments in which the score isassociated with a single student's answers, the evaluation engine 212can evaluate the student's performance in answering the questions. Insome embodiments in which the score information is associated with aclass, the evaluation engine 212 can evaluate the class's performance inanswering questions.

In some embodiments, information relating to a student's performance ora class's performance can be used by the evaluation engine to evaluatean educator, such as, for example, a teacher. In such an embodiment, theevaluation of the educator can be based, for example, on a comparison ofthe progress of the educator's students compared to other groups ofstudents or other metrics.

The evaluation engine 212 can receive information from the questionengine 212 and/or education engine 206 relating to the educator, subjectmatter, course work, topics, or subtopics associated with the questions.The evaluation engine 212 can further receive information from thequestion engine relating to circumstances in which the questions werereceived, such as, for example, time of day, month, location, orconditions existing at the location. Based on the received information,the evaluation engine 212 applies one or several threshold values to thescore to determine the user's proficiency in the subject matter, coursework, topics, or subtopics associated with the questions. The evaluationengine 212 can generate a report indicating the user's proficiency andcan provide this report to the network interface for communication toother components of the learning system.

The evaluation engine 212 can be software located on the user device102, or a portal, such as a web portal, accessible via the user device102. The evaluation engine 212 can also be located on another componentof the learning system 100 such as, for example, the managed learningenvironment, and accessible by the user device 102.

With reference to FIG. 3, a block diagram of an embodiment of the LMS108 is shown. As discussed above, the LMS 108 facilitates education byallowing the use of self-service and self-guided educational services soas to allow a student to progress at their own selected pace and toreview material as desired

The LMS 108 includes a customization system 302. The customizationsystem 302 adjusts the educational experience to match the student'seducational needs and desires. In some embodiments, the customizationsystem 302 can query the profile database 106-C for student preferences,including learning preferences. The customization engine canadditionally query the profile database 106-C for information relatingto the student's past performance to identify potential areas ofdifficulty with new subject matter. In some embodiments, thecustomization engine can further query the educator database 106-D foreducator input information relating to the student's needs. Thisinformation can be based on the educator's past experiences with thestudent and the strengths and weaknesses of the student. Based on thereceived information, the customization system 302 can modify theeducational material for the student.

The LMS 108 includes a student management system 304. The studentmanagement system 304 can track the student's progress through acurriculum, including one or several courses or trainings and can querythe profile database 106-C for information relating to the student'sprogress through the curriculum, including the student's educationalgoals, lessons that the student has completed, questions that thestudent has answered, and for results of the questions that the studenthas answered. The student management system 304 can additionally querythe content management system 110 to identify additional educationalinformation that the student can complete in order to complete thecurriculum and can additionally query the educator database 106-D foreducator inputs including the educator's educational goals for thestudent. Based on the received information, the student managementsystem 304 can modify the curriculum to maximize the student'seducational experience.

The LMS 108 includes a content delivery system 306. The content deliverysystem 306 provides content to the student device 102-A. The contentdelivery system 306 receives educational material from the contentmanagement system 110 and/or from the student management system 304 andprovides the material to the student device 102-A in the appropriateformat.

The LMS 108 includes a testing system 308 and an evaluation system 310.The testing system 308 queries the question database 106-F for questionsand associated answers. After receiving the questions and associatedanswers, the testing system 308 can transform the questions into atestable format by, for example, associating a student's and/oreducator's information with the questions or placing the questions in apage format. The testing system 308 can then provide the questions tothe student.

The testing system 308 can additionally receive responses from thestudent and determine if the received responses are correct. In someembodiments, the testing system 308 determines if the received responseare correct by querying the question database 106-F for answer data. Thetesting system 308 can then compare the user provided answers with theanswer data and determine whether the student answers are correct. Insome embodiments, this determination of the correctness of the studentprovided answers can be made according to a boolean-valued function. Thetesting system 308 can generate a boolean-value for all or some of thereceived answers. In some embodiments, a correct answer can be assigneda boolean-value of true, which can be represented by a value such as “1”and an incorrect answer can be assigned a boolean-value of false, whichcan be represented by a value such as “−1”.

The evaluation system 310 receives the corrected answers from thetesting system 308 and evaluates the answers for indicia of studentcomprehension. In some embodiments, the evaluation system 310 generatesa response metric indicative of the scatter, randomness, and or slope ofdata associated with a student's answers. The process used by theevaluation system 310 to evaluate the answers will be discussed ingreater detail below.

With reference now to FIG. 4, a swimlane flowchart of an embodiment of aprocess 400 for detecting a threshold of scatter or randomness inquestionnaire answer data is shown. The headers of the swimlanesidentify components of the learning system 100 that can perform andindicated step.

The process 400 begins at block 400, wherein the managed learningenvironment 104 receives the user data. The user data can be receivedfrom the profile database 106-C and/or the educator database 106-D andcan be received by the customization system 302 and/or the studentmanagement system 304.

In some embodiments, the user data can include student data, such as,the student profile, including student preferences or student educationand/or performance history. In some embodiments, the user data caninclude the educator profile, including, educator preferences oreducator performance history.

After receiving the user data, the process 400 proceeds to block 404,wherein the question data is received. The question data can be receivedfrom the question database 106-F and can be received by the managedlearning environment 104 and specifically by the testing system 308. Thequestion data can include one or several questions relating to one orseveral topics, and answers to the received questions.

After receiving the question data, the process 400 proceeds to block 406wherein the educator, for example via the teacher device 102-B, requestsquestioning for the student. In some embodiments, this request can bemade in response to a prompt by the student management system 304 or insome embodiments the request can be made by the student managementsystem 304.

After the request has been made, the process 400 proceeds to block 408wherein the questionnaire is generated. The questionnaire includes atleast one question and requests a student response to that question. Thequestionnaire can be, for example, an assignment, a quiz, or a test. Insome embodiments, the questionnaire can be a preexisting questionnaire,in which case, the generation of the questionnaire can include queryingthe question database 106-F for the completed questionnaire. In someembodiments, the questionnaire can be generated by the testing system308 of the LMS 108. The testing system 308 queries the question database106-F for questions and associated answers to provide to the student. Insome embodiments, the requested questions and answers are associatedwith one or several subject matters and/or topics. The testing system308 compiles the requested questions and answers into a questionnaire.

After the questionnaire has been generated, the process 400 proceeds toblock 410 and the testing system 308 of the LMS 108 sends thequestionnaire via the network 107 to the student device 102-A, whereinthe questionnaire is received.

After the questionnaire has been received, the student provides answerdata. In some embodiments, the user can provide answer data to thestudent device 102-A via the user interface 202 of the student device102-A. In some embodiments, the student can provide the answer data tothe teacher in a non-digital form, such as, for example, by writing theanswers, by completing a multiple choice answer sheet, or orally. Insuch embodiments, the educator can enter the answer data into thelearning system 100 via the teacher device 102-B or administrator device102-C. In some embodiments, a student may be allowed to answer aquestion multiple times, receiving feedback after each answer indicatingwhether the question is correctly answered, until the question iscorrectly answered. The answer data is sent from the network interface204 of the user device 102 to the managed learning system 104 via thenetwork 107.

After the answer data is provided, the process 400 proceeds to block414, wherein a boolean-value is generated for each answer. Inembodiments in which the student is allowed to answer a questionmultiple times until the question is correctly answered, theboolean-value may be generated for some of the submitted answers, suchas, for example, the first answer submitted by the student in responseto the question, called a first submitted answer, or for all of thesubmitted answers. The boolean-value corresponds to the correctness ofthe answer provided by the student. Thus, a correct answer can beassigned a boolean-value of true, which can, in some embodiments berepresented by “1”, and an incorrect answer can be assigned aboolean-value of false, which can, in some embodiments, be representedby “−1”. The boolean-value can be generated by the testing system 308.

After the boolean-value has been generated, the process 400 proceeds toblock 416 wherein a response metric is generated. The response metric isgenerated, in part, based on the generated boolean-values. In someembodiments, in which the assigned boolean-values are a “1” for acorrect answer and a “−1” for an incorrect answer, the response metriccan be based on the boolean-values and/or on the sum of theboolean-values. The response metric can be generated by the evaluationsystem 310 of the managed learning environment 104, and in someembodiments, response metric score can be generated by the scoregenerator 210. The generation of the response metric will be discussedin greater detail below.

After the response metric has been generated, the process 400 proceedsto decision state 418 wherein the learning system 100 determines whetherthe threshold is reached. In some embodiments, the determination ofwhether the threshold is reached is made by the evaluation engine 212 ofthe user device 102 or by the evaluation system 310 of the LMS 108 ofthe managed learning system 104.

If the threshold is not reached and the student has demonstrated anadequate level of comprehension of the topic(s) associated with thequestions, and the process proceeds to block 420 wherein the studentreceives an indication of success. In some embodiments, the indicationof success can be sent to, for example, at least one of the studentdevice 102-A, the teacher device 102-B, the administrator device 102-C,and/or the parent device 102-D. The indication of success can be sentvia the network 10 and can be received by the network interface 204.

If the threshold is reached and the student has demonstrated aninsufficient level of comprehension, the process 400 proceeds to block422 and the managed learning environment 104 sends an alert that isreceived at blocks 424 by the one or several user devices 102 at block422. The alert can include information identifying the student,identifying the questions that led to the alert, and/or identify thetopic associated with the questions.

In some embodiments, the alert can be sent via the network 107 andreceived by the network interface 204 of one or more of the user devices102. In some embodiments, the user device 102 receiving the alert canprovide the alert to the user and, in some embodiments, request a userinput in response to the alert, such as, for example, a confirmation ofreceipt.

After the alert is received, the process 400 proceeds to block 426,wherein remedial action is recommended. In some embodiments, the alertcan trigger the student management system 304 of the LMS 108, which canthen request remedial action, and in some embodiments, the alert canresult in the educator requesting remedial action. In some embodiments,information received with the alert can be used to identify one orseveral topics in which the student's comprehension level can beincreased. These topics can then form the basis of the requestedremedial action.

With reference now to FIG. 5, a flowchart of an embodiment of a process500 for detecting a threshold of scatter or randomness in questionnaireanswer data is shown.

The process 500 begins at block 502 wherein the databases 106 storequestion and user data. In some embodiments, the question data,including the questions and answers associated with those questions canbe generated by an educator using the authoring tool 112 and can be sentto the databases 106, and specifically to the question database 106-Fvia the network. The questions and answers associated with the questionscan then be stored in the databases 106.

In some embodiments, the questions and answers associated with thoseanswers can be uploaded to the learning system 100 via one of the userdevices 102 or via the managed learning environment 104 and can be sentto the databases 106, and specifically to the question database 106-Fvia the network. The questions and answers associated with the questionscan then be stored in the databases 106.

The user data can include student data, educator data, and parent data.The user data can be submitted from one of the user devices 102 or canbe generated based on a specific user's actions within the learningsystem 100. The user data can include past, current, and future courses,past testing results, preferences, past evaluations, and/or educationalgoals. The user information, whether submitted from a user device 102 orgenerated by the learning system can be stored in one or both of theprofile database 106-C and the educator database 106-D.

After the question and user data is stored, the process 500 proceeds toblock 504 wherein question data is sent. The question data is retrievedfrom the question database 106-F. After the question data is retrieved,the testing system 308 can transform the questions into a testableformat and the testing system 308. The questions can then be sent fromthe testing system 308 within the LMS 108 via the network 107 to thedesired user device 102, including, for example, the student device102-A, the teacher device 102-B, and/or the administrator device 102-C.

After the question data is sent, the process 500 proceeds to block 506wherein answer data is received. In some embodiments, the answer data isreceived, via the network 107, by the managed learning environment 104,by the LMS 108, and/or by the testing system 308. The answer data can besent from a user device 102, such as the student device 102-A, theteacher device 102-B, and/or the administrator device 102-C. In someembodiments, the answer data can be created on the user device 102 byuse of the authoring tool 112 and/or by use of the question engine 208.

After the answer data is received, the process 500 proceeds to block 508wherein the answer data is stored. The answer data is stored in thedatabases 106, and can be particularly stored in the question database106-F.

After the answer is stored, the process 500 proceeds to block 510wherein the correctness of the answers is determined. In someembodiments, the correctness of the answer is determined by the testingsystem 308 comparing the answer received from the student with thequestion answer. In some embodiments, the testing system 308 can querythe question database 106-F for the answer submitted by the student andfor the answer to the question. The testing system 308 receives thisinformation and then determines if the answer submitted by the studentmatches the answer to the question.

After the correctness of the answers is determined, the process 500proceeds to decision state 512 wherein the received answers are sortedbased on their correctness. This sorting is performed by the testingsystem 308. If the submitted answer is incorrect, a value indicative ofthe incorrectness of the submitted answer, such as, for example “−1”, isassociated with the answer, and the process 500 proceeds to block 514wherein the value indicative of the incorrect answer is stored.Returning again to decision state 512, if the submitted answer iscorrect, a value indicative of the correctness of the submitted answer,such as, for example “1”, is associated with the answer, and the process500 proceeds to block 514 wherein the value indicative of the correctanswer is stored. The value indicative of a correct or an incorrectanswer can be sent from the managed learning environment 104, andspecifically from the testing system 308 of the LMS 108 to the databases106, and specifically to the question database 106-F.

After the answer has been stored, the process 500 proceeds to block 518wherein a response metric is generated. The response metric representsthe degree of scatter, roughness, and/or randomness in the valuesassociated with the answers or the slope of the sum of the valuesassociated with the answers. Thus, in embodiments in which the responsemetric corresponds with randomness, roughness, or scatter of theanswers, such as if the student answers all of the questions correctly alow response metric would be generated. The response metric can becalculated by the evaluation system 310 and/or the testing system 308.In some embodiments in which aspects of the LMS 108 are operating on theuser device 102, the response metric can be calculated by the scoregenerator 210. The details of some embodiments of methods for thegeneration of the response metric will be discussed in greater detailbelow.

After the response metric has been generated, the process 500 proceedsto block 520 wherein reporting information is determined. The reportinginformation can include the evaluation of the student's performance,such as, for example, a traditional grade, a percentage of questionsanswered correctly, or a response metric, identification of the topicand/or subject matter of the questions, and/or identification of anyparameters relating to the answering of the questions such as the timeof day, the day of the week, the time of year, or the testingconditions. The reporting information can include informationidentifying an applicable threshold, and can additionally identify theintended recipients of the report. These recipients can include, forexample, the student, an educator such as the teacher or anadministrator, a parent, or any other desired information.

After the reporting information has been determined, the process 500proceeds to decision state 522 wherein it is determined if the thresholdhas been reached. The determination of whether the threshold has beenreached can be made by the evaluation system 310 or in embodiments inwhich components of the LMS 108 are located on the user device 102 thedetermination of whether the threshold has been reached can be made bythe evaluation engine 212. The determination of whether the thresholdhas been reached can be made by comparing the response metric to thedetermined threshold.

If the threshold has been reached, the process 500 proceeds to block 524wherein the information relating to the reached threshold is stored. Theinformation relating to the reached threshold can be sent from theevaluation system 310 or from the evaluation engine 212 to the databases106, and particularly to profile database 106-C for storage.

After the information relating to the reached threshold has been stored,or if it is determined in decision state 522 that the threshold has notbeen reached, the process 500 proceeds to block 526 wherein a report isgenerated. The report is generated by the evaluation system 310 or bythe evaluation engine 212. The report can identify the student,educators associated with the student, the course or curriculum relatingto the questions, the student's performance in answering the questions,and whether any remedial teaching or follow-up is required.

In some embodiments, the report can focus on a single student'sperformance, or on several students' performance. In some embodiments,the report can provide a teacher an overview of the performance of eachof the students in her class, or an overview of the performance of theclass. In some embodiments, the report can be a heat chart showingstudent performance distributions over the time of a course or training.

After the report has been generated, the process 50 proceeds to block528 wherein the report is sent. The report can be sent to the users ofthe learning system 100 via the network 107. The report can be sent fromthe evaluation system 310 of the LMS 108 of from the evaluation engine212 of the user device. The report can be sent to all designatedrecipients of the report.

With reference to FIG. 6, a flowchart of an embodiment of a process 600for storing question and user data is shown.

The process 600 is a sub process performed in block 502 of FIG. 5,wherein the question and user data is stored. The process 600 begins inblock 602 wherein the questions and answers to the questions are stored.As discussed above, the questions and answers to the questions can begenerated with the authoring tool 112 or the questions and the answersto the questions can be uploaded to the learning system 100 via one ofthe user devices 102 or via the managed learning environment 104. Thequestions and the answers to the questions can be sent to the databases106, and specifically to the question database 106-F via the network.The questions and answers associated with the questions can then bestored in the databases 106.

After the questions and answers to the questions are stored, the process600 proceeds to step 604, wherein the user profile is stored. Asdiscussed above, the user profile can be created by the user(s), or canbe created based on the actions of the user within the managed learningenvironment 104. The stored user profiles can include informationrelating to education that the user has received, past or current userperformance, past, present, or future user courses, past, present, orfuture user teachers, past, present, or future user administrators, userparents, user preferences, any user disability, or any other usercharacteristic. The use profile can be stored in the profile database106-C.

After the user profile is stored, the process 600 proceeds to block 606,wherein the threshold data is stored. The threshold data can includedata relating to a single threshold, or to a plurality of thresholds.The threshold can be a generic, default threshold common to the learningsystem 100, the managed learning environment 104, or the LMS 108, or thethreshold can be customized to a teacher, a program, a class, a gradelevel, an age, a student, an aspect of one or several student profiles,a topic, or any other desired metric. The threshold can be set by a usersuch as a student, and educator such as a teacher or an administrator,or a parent, or be preset. The threshold data can be stored in thethreshold database 106-A.

After the threshold data is stored, the process 600 proceeds to block608, and then proceeds to block 504 of FIG. 5.

With reference now to FIG. 7, an embodiment of a process 700 fordetermining reporting information is shown. The process 700 is a subprocess performed in block 520 of FIG. 5, wherein the reportinginformation is determined.

The process 700 begins at block 702 wherein the question topic isdetermined. The question topic can be determined by the managed learningenvironment 104, and specifically by the evaluation system 310 of theLMS 108. In embodiments in which components of the LMS are located onthe user device 102 the question topic can be determined by theevaluation engine 212. The evaluation system 310 can query the databases106, and specifically query the topic database 106-E and/or the questiondatabase 106-F for the information identifying the topic associated withthe questions. The evaluation system 310 receives the informationidentifying the topic associated with the questions and therebydetermines the question topic.

After the question topic is determined, the process 700 proceeds toblock 704 wherein the educator is determined. The educator can bedetermined by the managed learning environment 104, and specifically bythe evaluation system 310 of the LMS 108. In embodiments in whichcomponents of the LMS are located on the user device 102 the questiontopic can be determined by the evaluation engine 212. The evaluationsystem 310 can query the databases 106, and specifically query theeducator database 106-D and the question database 106-F for informationidentifying the educator associated with the question. The evaluationsystem 310 receives the information identifying the educator associatedwith the question, and thereby determines the educator.

After the educator is determined, the process 700 proceeds to block 706wherein the threshold value is determined. In some embodiments, thethreshold value is determined by retrieving a threshold value that isstored in the threshold database 106-A. In some embodiments in which asingle threshold value is stored in the threshold database 106-A,determining the threshold value is accomplished by selecting the singlethreshold value. In some embodiments in which multiple threshold valuesare stored in the threshold database 106-A, the determination of thethreshold value can include the process of selecting one of the multiplethreshold values. This process for selecting one of several thresholdvalues is discussed in greater detail below.

After the threshold value is determined, the process 700 proceeds toblock 708, and then proceeds to block 522 of FIG. 5.

With reference now to FIG. 8, an embodiment of a process 800 fordetermining a threshold value is shown. The process 800 is a sub processperformed in block 706 of FIG. 7, wherein the threshold value isdetermined.

The process 800 begins at decision state 802 wherein it is determinedwhether the retrieved threshold data includes a threshold value specificto the user. This determination of whether the threshold data includes athreshold value specific to the user can be made by the evaluationsystem 310, or in embodiments in which portions of the LMS 108 arelocated on the user device 102, by the evaluation engine 212. If thethreshold data includes a user threshold, then the process 800 proceedsto block 804, and then proceeds to block 708 of FIG. 7.

If the threshold data does not include a user threshold, then theprocess 800 proceeds to decision state 806 wherein it is determinedwhether the retrieved threshold data includes a threshold value specificto the topic associated with the questions. This determination ofwhether the threshold data includes a threshold value specific to thetopic associated with the questions can be made by the evaluation system310, or in embodiments in which portions of the LMS 108 are located onthe user device 102, by the evaluation engine 212. If the threshold dataincludes a topic threshold, then the process 800 proceeds to block 804,and then proceeds to block 708 of FIG. 7.

If the threshold data does not include a topic threshold, then theprocess 800 proceeds to decision state 808 wherein it is determinedwhether the retrieved threshold data includes a threshold value specificto the educator giving the questions. This determination of whether thethreshold data includes a threshold value specific to the educatorgiving the questions can be made by the evaluation system 310, or inembodiments in which portions of the LMS 108 are located on the userdevice 102, by the evaluation engine 212. If the threshold data includesan educator threshold, then the process 800 proceeds to block 804, andthen proceeds to block 708 of FIG. 7.

If the threshold data does not include an educator threshold, then theprocess 800 proceeds to block 810 wherein a default threshold isdetermined. This determination of the default threshold can be made bythe evaluation system 310, or in embodiments in which portions of theLMS 108 are located on the user device 102, by the evaluation engine212. After the default threshold is identified, the process 800 proceedsto block 804, and then proceeds to block 708 of FIG. 7.

If a default threshold is not identified, the learning system 100 canquery the user for a threshold value. Such a provided threshold valuecan be stored in the threshold database 106-A. After receiving the userprovided threshold value, the process 800 can proceed to block 804, andthen proceed to block 708 of FIG. 7.

If the threshold data includes a topic threshold, then the process 800proceeds to block 804, and then proceeds to block 708 of FIG. 7.

With reference now to FIG. 9, an embodiment of a process 900 fordetermining a response metric is shown. The process 900 is a sub processperformed in block 518 of FIG. 5, wherein the response metric isgenerated. The process 900 is performed by the evaluation system 310,the evaluation engine 212, and/or the score generator 210.

The process 900 begins at block 902 wherein a net-score of answers iscalculated. The net-score can be a sum of the boolean-values, such as,for example, “1” for a correct answer and “−1” for an incorrect answer,associated with the answers that the student has provided. The net-scoreincludes input based on the number of correct answers given by thestudent and the number of incorrect answers given by the student. Insome embodiments, the net-score can be statically calculated for afinite timeframe, and in other embodiments, the net-score can be adynamically calculated so as to be updated as the student providesadditional answers. In embodiments in which a boolean-value is assignedto each of the answers, the net-score can be the sum of the assignedboolean-values.

After the net-score is calculated, the process 900 proceeds to block 904wherein a defined subset of answer data is selected. In someembodiments, this subset of answer data can be, for example, a number ofanswers that can be sequentially given to the student and/orsequentially answered by the student. In some embodiments, the subset ofanswer data can be less than 4, answers, less than 5 answers, less than10 answers, less than 15 answers, less than 20 answers, less than 30answers, less than 50 answers, less than 100 answers, less than 200answers, less than 500 answers, or any other or intermediate number ofanswers. In some embodiments, the subset can be reformed as the studentprovides additional answers.

After the subset of data is selected and defined, the process 900proceeds to block 906, wherein the subset of answer data is convolvedwith the net-score. The convolving of the subset of answer data with thenet-score can include adding the answer assigned boolean-values of thesubset of answer data to the net-score.

After the subset of answer data is convolved with the net-score, theprocess 900 proceeds to block 908 wherein a metric is generated from theconvolved subset and the net-score. A variety of metrics can begenerated from the convolved subset and the net-score, including metricsindicative of the degree of scatter or randomness (i.e. between correctand incorrect answers) in the student provided answers, including, forexample, statistical scores, indicative of trends in the subset answerdata and/or in the convolved data.

After the score is generated, the process 900 proceeds to decision state910, wherein it is determine if additional response metrics should begenerated. In embodiments in which additional answers are being receivedfrom the student, or in which unscored data exists, the decision can bemade to generate additional scores. If it is determined to generateadditional response metrics, the process 900 returns to block 902 andproceeds through process 900 as described above.

If it is determined to not generate additional response metrics, theprocess 900 proceeds to block 912, and then proceeds to block 520 ofFIG. 5.

With reference now to FIGS. 10 and 11, embodiments of a process 1000 anda process 1100 a process for generating a response metric is shown. Theprocess 1000 and the process 1100 are sub processes performed in block908 of FIG. 9, wherein the score from the convolved subset of answerdata and the net-score is generated. The processes 1000 and 1100 areperformed by the evaluation system 310, the evaluation engine 212,and/or the score generator 210.

Referring now to FIG. 10, the process 1000 begins in block 1002 whereina randomness metric is calculated over the result of the convolvedsubset of answer data and the net-score. The randomness metric can beany metric that describes the randomness or roughness of the answerdata. In some embodiments, the randomness metric can comprise a fractaldimension, a Hausdorff dimension, a multifractal, a Hurst exponent orcoefficient, a Holder exponent or coefficient, a singularity spectrum,and/or a multifractal spectrum. The fractal dimension can be calculatedusing a variety of algorithms for generating a fractal dimension,including, for example, a madogram algorithm such as is disclosed in“Estimators of Fractal Dimension: Assessing the Roughness of Time Seriesand Spatial Data” by T Gneiting, H. {hacek over (S)}ev{hacek over(c)}ikova, D. B. Percival, University of Washington (Seattle) TechnicalReport No. 577, 2010, the entirety of which is incorporated herein byreference.

In some embodiments, the randomness metric and/or the fractal dimensioncan be compared to the threshold value. In such an embodiment, thethreshold could correspond to a fractal dimension of 2, of 1.8, of 1.6,of 1.3, of 1.2, of 1.1, or of any other or intermediate value. In someembodiments, a fractal dimension of 2 could indicate a high levelrandomness in the student's answers, and thereby indicate that thestudent does not comprehend the topics associated with the questions.Similarly, a fractal dimension of 1.8 or 1.6 can indicate high, albeitrelatively less, randomness in the student's answers, and thereby canindicate that the student does not completely comprehend the topicsassociated with the questions. In some embodiments, a fractal dimensionof 1.3, 1.2, or 1.1 can indicate a low level or randomness in thestudent's answers, which may corresponds, for example, to a satisfactorylevel of comprehension.

After the randomness metric is calculated, the process 1000 proceeds toblock 1004, wherein the midpoint of the defined subset of thequestionnaire answer data is determined. The midpoint can be determinedusing a variety of techniques, including finding the middle number in asequence of student submitted answers.

After the midpoint of the defined subset of the questionnaire answerdata is determined, the process 1000 proceeds to block 1006, wherein thevalue of the randomness metric is assigned to the midpoint of thedefined subset of questionnaire answer data. After the value of therandomness metric is assigned to the midpoint, the process 1000 proceedsto block 1008, and then proceeds to block 910 of FIG. 9.

Referring now to FIG. 11, the process 1100 begins in block 1102 whereina slope of the net-score is calculated over the defined subset of answerdata. The slope can be calculated using any number of known techniques.

In some embodiments, the slope can be compared to the threshold value.In such an embodiment, the threshold could correspond to a negativeslope, a positive slope, or any other slope of the net-score over thedefined subset of answer data. In some embodiments, a negative slopeover the subset of questionnaire answer data, can correspond to a seriesof incorrect answers, and thereby indicate an unsatisfactory level ofcomprehension by the student. Similarly, in some embodiments, a positiveslope over the subset of questionnaire answer data, can correspond to aseries of correct answers, and thereby indicate a satisfactory level ofcomprehension by the student.

After the slope is calculated, the process 1100 proceeds to block 1104,wherein the midpoint of the defined subset of the questionnaire answerdata is determined. The midpoint can be determined using a variety oftechniques, including finding the middle number in a sequence of studentsubmitted answers.

After the midpoint of the defined subset of the questionnaire answerdata is determined, the process 1100 proceeds to block 1106, wherein thevalue of the slope is assigned to the midpoint of the defined subset ofquestionnaire answer data. After the value of the slope is assigned tothe midpoint, the process 1100 proceeds to block 1108, and then proceedsto block 910 of FIG. 9.

With reference now to FIG. 12, an exemplary environment with whichembodiments may be implemented is shown with a computer system 1200 thatcan be used by a user 1204 as all or a component of a learning system100. The computer system 1200 can include a computer 1202, keyboard1222, a network router 1212, a printer 1208, and a monitor 1206. Themonitor 1206, processor 1202 and keyboard 1222 are part of a computersystem 1226, which can be a laptop computer, desktop computer, handheldcomputer, mainframe computer, etc. The monitor 1206 can be a CRT, flatscreen, etc.

A user 1204 can input commands into the computer 1202 using variousinput devices, such as a mouse, keyboard 1222, track ball, touch screen,etc. If the computer system 1200 comprises a mainframe, a designer 1204can access the computer 1202 using, for example, a terminal or terminalinterface. Additionally, the computer system 1226 may be connected to aprinter 1208 and a server 1210 using a network router 1212, which mayconnect to the Internet 1218 or a WAN.

The server 1210 may, for example, be used to store additional softwareprograms and data. In one embodiment, software implementing the systemsand methods described herein can be stored on a storage medium in theserver 1210. Thus, the software can be run from the storage medium inthe server 1210. In another embodiment, software implementing thesystems and methods described herein can be stored on a storage mediumin the computer 1202. Thus, the software can be run from the storagemedium in the computer system 1226. Therefore, in this embodiment, thesoftware can be used whether or not computer 1202 is connected tonetwork router 1212. Printer 1208 may be connected directly to computer1202, in which case, the computer system 1226 can print whether or notit is connected to network router 1212.

With reference to FIG. 13, an embodiment of a special-purpose computersystem 1304 is shown. The above methods may be implemented bycomputer-program products that direct a computer system to perform theactions of the above-described methods and components. Each suchcomputer-program product may comprise sets of instructions (codes)embodied on a computer-readable medium that directs the processor of acomputer system to perform corresponding actions. The instructions maybe configured to run in sequential order, or in parallel (such as underdifferent processing threads), or in a combination thereof. Afterloading the computer-program products on a general purpose computersystem 1226, it is transformed into the special-purpose computer system1304.

Special-purpose computer system 1304 comprises a computer 1202, amonitor 1206 coupled to computer 1202, one or more additional useroutput devices 1330 (optional) coupled to computer 1202, one or moreuser input devices 1340 (e.g., keyboard, mouse, track ball, touchscreen) coupled to computer 1202, an optional communications interface1350 coupled to computer 1202, a computer-program product 1305 stored ina tangible computer-readable memory in computer 1202. Computer-programproduct 1305 directs system 1304 to perform the above-described methods.Computer 1202 may include one or more processors 1360 that communicatewith a number of peripheral devices via a bus subsystem 1390. Theseperipheral devices may include user output device(s) 1330, user inputdevice(s) 1340, communications interface 1350, and a storage subsystem,such as random access memory (RAM) 1370 and non-volatile storage drive1380 (e.g., disk drive, optical drive, solid state drive), which areforms of tangible computer-readable memory.

Computer-program product 1305 may be stored in non-volatile storagedrive 1380 or another computer-readable medium accessible to computer1202 and loaded into memory 1370. Each processor 1360 may comprise amicroprocessor, such as a microprocessor from Intel® or Advanced MicroDevices, Inc.®, or the like. To support computer-program product 1305,the computer 1202 runs an operating system that handles thecommunications of product 1305 with the above-noted components, as wellas the communications between the above-noted components in support ofthe computer-program product 1305. Exemplary operating systems includeWindows® or the like from Microsoft® Corporation, Solaris® from Oracle®,LINUX, UNIX, and the like.

User input devices 1340 include all possible types of devices andmechanisms to input information to computer system 1202. These mayinclude a keyboard, a keypad, a mouse, a scanner, a digital drawing pad,a touch screen incorporated into the display, audio input devices suchas voice recognition systems, microphones, and other types of inputdevices. In various embodiments, user input devices 1340 are typicallyembodied as a computer mouse, a trackball, a track pad, a joystick,wireless remote, a drawing tablet, a voice command system. User inputdevices 1340 typically allow a user to select objects, icons, text andthe like that appear on the monitor 1206 via a command such as a clickof a button or the like. User output devices 1330 include all possibletypes of devices and mechanisms to output information from computer1202. These may include a display (e.g., monitor 1206), printers,non-visual displays such as audio output devices, etc.

Communications interface 1350 provides an interface to othercommunication networks and devices and may serve as an interface toreceive data from and transmit data to other systems, WANs and/or theInternet 1218. Embodiments of communications interface 1350 typicallyinclude an Ethernet card, a modem (telephone, satellite, cable, ISDN), a(asynchronous) digital subscriber line (DSL) unit, a FireWire®interface, a USB® interface, a wireless network adapter, and the like.For example, communications interface 1350 may be coupled to a computernetwork, to a FireWire® bus, or the like. In other embodiments,communications interface 1350 may be physically integrated on themotherboard of computer 1202, and/or may be a software program, or thelike.

RAM 1370 and non-volatile storage drive 1380 are examples of tangiblecomputer-readable media configured to store data such ascomputer-program product embodiments of the present invention, includingexecutable computer code, human-readable code, or the like. Other typesof tangible computer-readable media include floppy disks, removable harddisks, optical storage media such as CD-ROMs, DVDs, bar codes,semiconductor memories such as flash memories, read-only-memories(ROMs), battery-backed volatile memories, networked storage devices, andthe like. RAM 1370 and non-volatile storage drive 1380 may be configuredto store the basic programming and data constructs that provide thefunctionality of various embodiments of the present invention, asdescribed above.

Software instruction sets that provide the functionality of the presentinvention may be stored in RAM 1370 and non-volatile storage drive 1380.These instruction sets or code may be executed by the processor(s) 1360.RAM 1370 and non-volatile storage drive 1380 may also provide arepository to store data and data structures used in accordance with thepresent invention. RAM 1370 and non-volatile storage drive 1380 mayinclude a number of memories including a main random access memory (RAM)to store of instructions and data during program execution and aread-only memory (ROM) in which fixed instructions are stored. RAM 1370and non-volatile storage drive 1380 may include a file storage subsystemproviding persistent (non-volatile) storage of program and/or datafiles. RAM 1370 and non-volatile storage drive 1380 may also includeremovable storage systems, such as removable flash memory.

Bus subsystem 1390 provides a mechanism to allow the various componentsand subsystems of computer 1202 communicate with each other as intended.Although bus subsystem 1390 is shown schematically as a single bus,alternative embodiments of the bus subsystem may utilize multiple bussesor communication paths within the computer 1202.

A number of variations and modifications of the disclosed embodimentscan also be used. Specific details are given in the above description toprovide a thorough understanding of the embodiments. However, it isunderstood that the embodiments may be practiced without these specificdetails. For example, well-known circuits, processes, algorithms,structures, and techniques may be shown without unnecessary detail inorder to avoid obscuring the embodiments.

Implementation of the techniques, blocks, steps and means describedabove may be done in various ways. For example, these techniques,blocks, steps and means may be implemented in hardware, software, or acombination thereof. For a hardware implementation, the processing unitsmay be implemented within one or more application specific integratedcircuits (ASICs), digital signal processors (DSPs), digital signalprocessing devices (DSPDs), programmable logic devices (PLDs), fieldprogrammable gate arrays (FPGAs), processors, controllers,micro-controllers, microprocessors, other electronic units designed toperform the functions described above, and/or a combination thereof.

Also, it is noted that the embodiments may be described as a processwhich is depicted as a flowchart, a flow diagram, a swim diagram, a dataflow diagram, a structure diagram, or a block diagram. Although adepiction may describe the operations as a sequential process, many ofthe operations can be performed in parallel or concurrently. Inaddition, the order of the operations may be re-arranged. A process isterminated when its operations are completed, but could have additionalsteps not included in the figure. A process may correspond to a method,a function, a procedure, a subroutine, a subprogram, etc. When a processcorresponds to a function, its termination corresponds to a return ofthe function to the calling function or the main function.

Furthermore, embodiments may be implemented by hardware, software,scripting languages, firmware, middleware, microcode, hardwaredescription languages, and/or any combination thereof. When implementedin software, firmware, middleware, scripting language, and/or microcode,the program code or code segments to perform the necessary tasks may bestored in a machine readable medium such as a storage medium. A codesegment or machine-executable instruction may represent a procedure, afunction, a subprogram, a program, a routine, a subroutine, a module, asoftware package, a script, a class, or any combination of instructions,data structures, and/or program statements. A code segment may becoupled to another code segment or a hardware circuit by passing and/orreceiving information, data, arguments, parameters, and/or memorycontents. Information, arguments, parameters, data, etc. may be passed,forwarded, or transmitted via any suitable means including memorysharing, message passing, token passing, network transmission, etc.

For a firmware and/or software implementation, the methodologies may beimplemented with modules (e.g., procedures, functions, and so on) thatperform the functions described herein. Any machine-readable mediumtangibly embodying instructions may be used in implementing themethodologies described herein. For example, software codes may bestored in a memory. Memory may be implemented within the processor orexternal to the processor. As used herein the term “memory” refers toany type of long term, short term, volatile, nonvolatile, or otherstorage medium and is not to be limited to any particular type of memoryor number of memories, or type of media upon which memory is stored.

Moreover, as disclosed herein, the term “storage medium” may representone or more memories for storing data, including read only memory (ROM),random access memory (RAM), magnetic RAM, core memory, magnetic diskstorage mediums, optical storage mediums, flash memory devices and/orother machine readable mediums for storing information. The term“machine-readable medium” includes, but is not limited to portable orfixed storage devices, optical storage devices, and/or various otherstorage mediums capable of storing that contain or carry instruction(s)and/or data.

While the principles of the disclosure have been described above inconnection with specific apparatuses and methods, it is to be clearlyunderstood that this description is made only by way of example and notas limitation on the scope of the disclosure.

What is claimed is:
 1. A method for determining and reacting toquestionnaire response patterns, the method comprising: storing inmemory electronic data comprising questions and answers to thequestions, wherein the questions are associated with a common topic;storing in the memory a user profile comprising data identifying a userand data relating to the user's past performance in answering questions;storing in the memory data defining a threshold value, wherein thethreshold value comprises at least one of: a value associated with theuser; a value associated with the topic; and a generic value; receivingat a processor questionnaire answer data, wherein the questionnaireanswer data comprises user provided answers to the questions;determining with the processor a user associated with the questionnaireanswer data; storing in the memory the questionnaire answer data in anelectronic store; determining with the processor whether thequestionnaire answers are correct or incorrect according to aboolean-valued function; storing in the memory a boolean valueindicating a correct answer for correct user provided answers; storingin the memory a boolean value indicating an incorrect answer forincorrect user provided answers; performing a function on theboolean-value outcome to generate a response metric indicative of ascatter or randomness pattern of the questionnaire answer data; whereingenerating the response metric indicative of the scatter or randomnesspattern of the questionnaire answer data comprises: calculating anet-score of correct answers; selecting a temporally defined subset ofthe questionnaire answer data; convolving the subset with the net-score;and generating a metric from the convolved subset and the net-score;wherein the metric indicates the level of scatter or randomness of thequestionnaire answer data; determining with the processor an educatorfor the user and the common topic associated with the questions, whereinthe educator supervises the user; comparing with the processor theresponse metric indicative of the scatter or randomness of thequestionnaire answer data with the data defining the threshold value;sending a message to an educator identifying the user, the topic, andthat the threshold value has been reached; and storing in the memorydata in the user profile identifying the topic and indicating that thethreshold value has been reached.
 2. The method for determining andreacting to questionnaire response patterns as recited in claim 1,wherein the questionnaire answer data is received from a student deviceand/or an educator device.
 3. The method for determining and reacting toquestionnaire response patterns as recited in claim 1, furthercomprising determining a threshold value, wherein determining thethreshold value comprises determining whether the user, the commontopic, and/or the educator is assigned a predetermined threshold thatdefines the threshold value.
 4. The method for determining and reactingto questionnaire response patterns as recited in claim 1, wherein thethreshold value is reached when the response metric indicates more thanan allowable amount of scatter or randomness pattern in thequestionnaire answer data.
 5. The method for determining and reacting toquestionnaire response patterns as recited in claim 4, whereingenerating a metric from the convolved subset and the net-scorecomprises generating a fractal dimension.
 6. The method for determiningand reacting to questionnaire response patterns as recited in claim 1,wherein generating the response metric indicative of the scatter orrandomness pattern of the questionnaire answer data comprises:calculating a net-score of correct answers; selecting a temporallydefined subset of the questionnaire answer data; and calculating theslope of the net-score in the temporally defined subset of questionnairedata.
 7. The method for determining and reacting to questionnaireresponse patterns as recited in claim 1, wherein the message comprisesidentifying learning material for the student.
 8. The method fordetermining and reacting to questionnaire response patterns as recitedin claim 1, wherein the questionnaire answer data comprises data for thefirst submitted answers.
 9. A learning system for determining andreacting to questionnaire response patterns, the learning systemcomprising: one or more processors; and one or more non-transitorycomputer readable media comprising instructions executable by the one ormore processors that direct the one or more processors to: storeelectronic data comprising questions and answers to the questions,wherein the questions are associated with a common topic; store a userprofile comprising data identifying a user and data relating to theuser's past performance in answering questions; store data defining athreshold value, wherein the threshold value comprises at least one of:a value associated with the user; a value associated with the topic; anda generic value; receive questionnaire answer data, wherein thequestionnaire answer data comprises user provided answers to questions;determine a user associated with the questionnaire answer data; storethe questionnaire answer data in an electronic store; determine whetherthe answers are correct or incorrect according to a boolean-valuedfunction; store a boolean value indicating a correct answer for correctuser provided answers; store a boolean value indicating an incorrectanswer for incorrect user provided answers; perform a function on theboolean-value outcome to generate a response metric indicative of ascatter or randomness pattern of the questionnaire answer data; whereingenerating the response metric indicative of the scatter or randomnesspattern of the questionnaire answer data comprises: calculating anet-score of correct answers; selecting a temporally defined subset ofthe questionnaire answer data; convolving the subset with the net-score;and generating a metric from the convolved subset and the net-score;wherein the metric indicates the level of scatter or randomness of thequestionnaire answer data; determine an educator for the user and thecommon topic associated with the questions, wherein the educatorsupervises the user; compare the response metric indicative of thescatter or randomness pattern of the questionnaire answer data with datadefining a threshold value; send a message to an educator identifyingthe user, the topic, and that the threshold value has been reached; andstore data in the user profile identifying the topic and indicating thatthe threshold value has been reached.
 10. The learning system fordetermining and reacting to questionnaire response patterns recited inclaim 9, wherein the questionnaire answer data is received from astudent device and/or an educator device.
 11. The learning system fordetermining and reacting to questionnaire response patterns recited inclaim 9, wherein the instructions executable by the one or moreprocessors direct the one or more processors to determine a thresholdvalue, wherein determining the threshold value comprises determiningthat the user, the common topic, or the educator has a predeterminedthreshold that is used to define the threshold value.
 12. The learningsystem for determining and reacting to questionnaire response patternsrecited in claim 9, wherein the threshold value is reached when theresponse metric indicates more than an allowable amount of scatter orrandomness pattern in the questionnaire answer data.
 13. The learningsystem for determining and reacting to questionnaire response patternsrecited in claim 12, wherein generating a metric from the convolvedsubset and the net-score comprises generating a fractal dimension. 14.The learning system for determining and reacting to questionnaireresponse patterns recited in claim 9, wherein generating the responsemetric indicative of the scatter or randomness pattern of thequestionnaire answer data comprises: calculating a net-score of correctanswers; selecting a temporally defined subset of the questionnaireanswer data; and calculating the slope of the net-score in thetemporally defined subset of questionnaire data.
 15. The learning systemfor determining and reacting to questionnaire response patterns recitedin claim 9, wherein the message identifies learning material for thestudent.
 16. A method for generating a report in response to determiningto questionnaire response patterns, the method comprising: receiving ata processor question data, wherein the question data comprisesquestions, answers associated with the questions, and a topic associatedwith the questions; receiving at the processor user data, wherein theuser data comprises user identification and user performance history,wherein the user performance history indicates the number of questionsthat the user has received and the number of questions that the user hascorrectly answered; creating at the processor a questionnaire based onthe question data; sending the questionnaire; receiving at the processoranswer data comprising submitted responses to the questions in thequestionnaire; determining at the processor according to aboolean-valued function the correctness of the answers, wherein correctanswers are assigned a first boolean-value and incorrect answers areassigned a second boolean-value; generating at the processor a responsemetric based on the boolean-values assigned to the answers, wherein theresponse metric provides an indicator of the degree of a scatter orrandomness pattern in the answers, wherein generating the responsemetric indicative of the scatter or randomness pattern of thequestionnaire answer data comprises: calculating a net-score of correctanswers; selecting a temporally defined subset of the questionnaireanswer data; convolving the subset with the net-score; and generating ametric from the convolved subset and the net-score; wherein the metricindicates the level of scatter or randomness of the questionnaire answerdata; evaluating at the processor the submitted responses based on theresponse metric; storing in memory data indicating degree of the scatteror randomness pattern in the answers; generating at the processor areport of the results of the evaluation, wherein generating the reportcomprises: identifying the recipients of the report; and determining theuser associated with the report; and sending the report to theidentified recipients.
 17. The method for generating the report inresponse to determining to questionnaire response patterns as recited inclaim 16, further comprising determining a threshold value, whereindetermining the threshold value comprises determining that the user, thecommon topic, and/or the educator has a predetermined threshold that isused to define the threshold value.
 18. The method for generating thereport in response to determining to questionnaire response patterns asrecited in claim 16, wherein the answer data comprises responsessubmitted to questions by a plurality of users.
 19. The method forgenerating the report in response to determining to questionnaireresponse patterns as recited in claim 18, wherein generating the reportfurther comprises identifying a subgroup of the plurality of usershaving a common course.
 20. The method for generating the report inresponse to determining to questionnaire response patterns as recited inclaim 19, wherein generating the report further comprises aggregatingthe data indicating the degree of the scatter or randomness pattern inthe answers for the plurality of users having the common course.
 21. Amethod for determining and reacting to questionnaire response patterns,the method comprising: storing in memory electronic data comprisingquestions and answers to the questions associated with a common topic;storing in the memory data defining a threshold value, wherein thethreshold value comprises at least one of: a value associated with auser; a value associated with the topic; and a generic value; receivingquestionnaire answers, wherein the questionnaire answers comprise userprovided answers to the questions; assigning with a processor aboolean-value to each of a plurality of the received questionnaireanswers, the boolean-value indicating whether the questionnaire answerwas correct or incorrect; performing on the processor a function on theassigned boolean-values to generate a response metric indicative of thescatter or randomness pattern of the questionnaire answer data, whereingenerating the response metric indicative of the scatter or randomnesspattern of the questionnaire answer data comprises: calculating anet-score of correct answers; selecting a temporally defined subset ofthe questionnaire answer data; convolving the subset with the net-score;and generating a metric from the convolved subset and the net-score;wherein the metric indicates the level of scatter or randomness of thequestionnaire answer data; comparing with the processor the responsemetric indicative of the scatter or randomness pattern of thequestionnaire answer data with the data defining the threshold value;and generating a message identifying the user providing thequestionnaire answers, the topic, and that the threshold value has beenreached.
 22. The method for determining and reacting to questionnaireresponse patterns as recited in claim 21, wherein the threshold value isreached when the response metric indicates more than an allowable amountof scatter or randomness pattern in the questionnaire answer data. 23.The method for determining and reacting to questionnaire responsepatterns as recited in claim 22, wherein generating a metric from theconvolved subset and the net-score comprises generating a fractaldimension.
 24. The method for determining and reacting to questionnaireresponse patterns as recited in claim 21, wherein generating theresponse metric indicative of the scatter or randomness pattern of thequestionnaire answer data comprises: calculating a net-score of correctanswers; selecting a temporally defined subset of the questionnaireanswer data; and calculating the slope of the net-score in thetemporally defined subset of questionnaire data.
 25. The method fordetermining and reacting to questionnaire response patterns as recitedin claim 21, further comprising determining a threshold value, whereindetermining the threshold value comprises determining whether the user,the common topic, and/or the educator is assigned a predeterminedthreshold that defines the threshold value.